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FOREWORD 


The  ARI  Enlistment  Decision  Surveys  have  become  an  impor¬ 
tant  source  of  information  for  U.S.  Army  policy  makers  and 
planners.  Originally  developed  by  the  Army  Research  Institute 
in  1982  under  a  commission  from  the  Deputy  Chief  of  Staff  for 
Personnel,  the  1987  USAREC  New  Recruit  Survey  is  the  eighth  of 
a  series  of  research  projects  investigating  new  U.S.  Army  re¬ 
cruits.  Previous  surveys  conducted  in  this  series  include 
(1)  Summer  1982;  (2)  Summer  1983;  (3)  Winter  1983;  (4)  Summer 

1984;  (5)  Winter  1984;  (6)  Summer  1985;  and  (7)  Summer  1986. 

This  multiyear  endeavor  was  conducted  to  measure  the  en¬ 
listment  motivations,  attitudes,  knowledge,  and  personal  char¬ 
acteristics  of  new  recruits  at  the  time  of  their  initial  entry 
into  the  U.S.  Army.  The  ability  to  track  changes  in  new  re¬ 
cruit  characteristics  over  time,  as  well  as  to  provide  an  op¬ 
portunity  for  the  collection  of  new,  policy-relevant  informa¬ 
tion  during  each  survey  administration,  has  provided  an  impetus 
for  continuing  the  survey  efforts. 

The  Army  Research  Institute's  participation  in  this  effort 
is  part  of  an  on-going  research  program  designed  to  enhance  the 
quality  of  Army  personnel.  This  work  is  an  essential  part  of 
the  mission  of  API's  Manpower  and  Personnel  Policy  Research 
Group  to  conduct  research  to  improve  the  Army's  capability  to 
effectively  and  efficiently  recruit  its  personnel.  Both  the 
ARI  New  Recruit  Surveys  of  1982  and  1983  were  conducted  at  the 
request  of  the  U.S.  Army  Deputy  Chief  of  Staff  for  Personnel. 
Sponsorship  of  the  1984,  1985,  and  1986  surveys  was  assumed  by 

the  U.S.  Army  Recruiting  Command,  with  the  continuing  interest 
of  the  Office  of  the  Deputy  Chief  of  Staff  for  Personnel .  The 
Summer  1987  survey  marked  the  first  administration  of  the  sur¬ 
vey  conducted  entirely  under  the  control  of  the  U.S.  Army  Re¬ 
cruiting  Command.  Results  in  this  report  were  briefed  to  the 
Chief,  USAREC  Advertising  Research  and  Analysis,  on  22  March 
1988. 
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Technical  Director 


A  COMPARISON  OF  SURVEY  LENGTH  WITH  READING  DIFFICULTY  LEVEL  AND 
SELECTED  RECRUIT  CHARACTERISTICS  FOR  THE  1987  USAREC  SURVEY 
OF  ACTIVE  ARMY  RECRUITS 


EXECUTIVE  SUMMARY 


Requirement: 

To  ensure  the  quality  of  survey  data  of  U.S.  Army  recruit 
respondents  who  provide  information  on  enlistment  motivations, 
attitudes,  knowledge,  and  demographic  characteristics  at  the 
time  of  initial  entry  into  the  service. 


Procedure: 

Analyses  were  conducted  using  data  from  2,998  respondents 
to  the  1987  USAREC  New  Recruit  Survey  (NRS-87)  administered  at 
the  eight  U.S.  Army  Reception  Battalions  during  June,  July,  and 
August  of  1987.  Specifically,  reading  difficulty  levels  of  the 
survey  booklets  were  ascertained,  life  table  analyses  were  con¬ 
ducted  to  determine  at  what  point  different  target  group  mem¬ 
bers  failed  to  complete  the  survey  items,  and  analyses  of  vari¬ 
ance  were  completed  to  determine  the  factors  within  target 
groups  that  most  affected  survey  completion  rates. 


Findings: 

Respondents  taking  the  1987  USAREC  New  Recruit  Survey 
needed  a  beginning  college  reading  level  to  understand  and  re¬ 
spond  well  to  the  entire  sec  of  survey  items.  Reading  diffi¬ 
culty  levels  increase  as  respondents  move  through  the  survey's 
section,  ranging  in  difficulty  from  the  sixth  grade  to  the 
first  year  of  college  levels.  Approximately  20%  of  all  re¬ 
spondents  failed  to  complete  the  survey  questionnaires.  A 
direct  relationship  exists  between  AFQT  Test  Category  and 
survey  completion.  Test  Category  I  respondents  completed  the 
survey  most  often,  followed  in  order  by  those  individuals  in 
Categories  II,  IIIA,  IIIB,  and  IV.  Future  versions  of  the 
survey  should  be  written  at  the  lowest  readability  level  that 
can  still  present  the  content  of  the  questions.  A  recommended 
level  is  the  sixth  grade  for  maximum  assurance  that  respondents 
will  be  able  to  read  and  comprehend  the  survey  items.  In  addi¬ 
tion  to  reducing  the  reading  level  of  the  instrument,  a  maximum 
survey  length  of  150  items  (or  12  OPSCAN  instrument  pages)  is 
encouraged  to  ensure  maximum  completion  rates  by  all  survey 
respondents. 


vii 


Utilization  of  Findings: 

The  USAREC  New  Recruit  Surveys  will  continue  to  provide 
information  to  U.S.  Army  policy  makers  and  personnel  planners 
about  the  quantity  and  quality  of  U.S.  Army  accessions.  Im¬ 
proving  the  item  response  abilities  of  all  respondents  will 
help  to  ensure  that  the  maximum  individual  representation  and 
precision  of  response  is  achieved  without  burdening  the  sol¬ 
diers  completing  the  questionnaire.  Results  of  this  research 
will  enable  planners  to  evaluate  the  design  of  subsequent 
USAREC  New  Recruit  Surveys  in  order  to  continue  providing 
useful  and  unbiased  information  about  recruits. 
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A  COMPARISON  OF  SURVEY  LENGTH  WITH  READING  DIFFICULTY 
LEVEL  AND  SELECTED  RECRUIT  CHARACTERISTICS  FOR 
THE  1987  USAREC  SURVEY  OF  ACTIVE  ARMY  RECRUITS 


INTRODUCTION 

This  report  is  provided  as  a  source  of  background 
documentation  for  current  and  potential  users'  of  the  data 
collected  through  the  1986  U.S.  Army  Research  Institute's  (ARI) 
New  Recruit  Survey  (NRS-86)  and  the  1987  US  Army  Recruiting 
Command's  (USAREC)  New  Recruit  Survey.  This  document  examines 
the  background  history  of  the  New  Recruit  Survey  administrations, 
and  compares  the  survey  instrument  type  and  length  across  the 
years.  In  addition  to  providing  background  information,  this 
document  also  provides  information  that  should  be  of  use  to 
survey  developers  of  future  NRS-type  instruments.  Reading 
difficulty  levels  are  computed  for  the  1986  and  1987  New  Recruit 
Survey  questionnaires.  Using  data  collected  during  the  Summer, 
1987  administration  of  the  USAREC  New  Recruit  Survey,  test  length 
considerations  are  examined  by  analyzing  characteristics  of 
those  respondents  who  failed  to  complete  the  entire  survey  with 
those  respondents  who  did  answer  all  items.  Recommendations  are 
made  for  future  versions  of  the  survey. 

NRS  Project  Background 

The  ARI  Enlistmeiit  Decision  Surveys  have  become  an 
important  source  of  information  to  U.S.  Army  policy  makers  and 
planners.  Originally  developed  by  ARI  in  1982  under  a  commission 
by  the  Deputy  Chief  of  Staff  for  Personnel,  the  1987  USAREC  New 
Recruit  Survey  is  the  eighth  of  a  series  of  survey  research 
projects  investigating  new  U.S.  Army  recruits.  Previous  survey 
research  projects  conducted  in  this  series  include:  (1)  Summer, 
1982;  (2)  Summer,  1983;  (3)  Winter,  1983;  (4)  Summer,  1984;  (5) 
Winter,  1984;  (6)  Summer,  1985;  and  (7)  Summer,  1986. 

This  multi-year  survey  research  endeavor  has  been  conducted 
to  measure  the  enlistment  motivations,  attitudes,  knowledge,  and 
personal  characteristics  of  new  recruits  at  the  time  of  their 
initial  entry  into  the  U.S.  Army.  The  ability  to  track  changes 
in  new  recruit  characteristics  over  time,  as  well  as  to  provide 
an  opportunity  for  the  collection  of  new,  policy-relevant 
information  during  each  survey  administration  has  provided  an 
impetus  for  continuing  the  survey  efforts. 

Both  the  ARI  New  Recruit  Surveys  of  1982  and  1983  were 
conducted  at  the  request  of  the  U.S.  Army  Deputy  Chief  of  Staff 
for  Personnel.  Sponsorship  of  the  1984,  1985,  and  1986  surveys 
was  assumed  by  the  U.S.  Army  Recruiting  Command,  with  the 
continuing  interest  of  the  Office  of  the  Deputy  Chief  of  Staff 
for  Personnel.  The  Summer,  1987  survey  marked  the  first 


administration  of  the  survey  conducted  entirely  under  the 
control  of  the  U.S.  Army  Recruiting  Command.  Although  the 
sponsorship  and  specific  question  content  has  changed  across 
time,  the  basic  charter  for  these  field  research  surveys  has 
remained  unchanged: 

•  Determine  who  is  enlisting  in  the  U.S.  Army  and  why. 

•  Determine  how  to  target  recruiting  resources  to 
attract  high  quality  recruits. 

•  Determine  why  recent  recruits  joined  and  their 
propensity  to  remain  in  the  service. 

•  Determine  which  recruiting  and  advertising  practices 
are  proving  the  most  effective  and  why. 

In  addition  to  addressing  these  specific  objectives,  the 
survey  efforts  have  expanded  the  availability  of  research  data  by 
which  U.S.  Army  enlistment  and  reenlistment  decision  processes 
can  be  modeled. 


SURVEY  INSTRUMENT  DEVELOPMENT 

Background 

The  1986  and  1987  NRS  were  the  first  New  Recruit  Surveys  to 
make  use  of  optically  scannable  survey  booklets.  The  use  of 
such  booklets  in  the  collection  of  survey-related  information 
requires  that  the  process  of  survey  design  and  layout  be 
approached  with  great  care  and  patience.  The  design,  layout, 
typesetting,  keylining,  and  printing  is  accomplished  using 
methods  familiar  to  persons  in  the  printing  industry.  Additional 
requirements  that  are  unique  to  Optical  Mark  Recognition  (OMR) 
scanning  systems  precludes  the  use  of  instant  print  or  offset 
print  shops  that  are  commonly  used  to  produce  standard  office 
forms.  The  process  by  which  the  survey  content  was  developed  and 
transformed  into  optically  scannable  booklets  is  presented  in  the 
following  sections. 

Survey  Content  Development 

With  minor  exceptions,  the  format  and  content  of  the  1986 
and  1987  NRS  are  identical.  The  same  booklet  layout,  question 
content,  and  item  location  were  used  in  each  version.  Thus,  the 
development  steps  that  follow  are  the  same  for  both  surveys.  The 
development  of  the  survey  questionnaire  content  involved  a 
three-stage  review  and  approval  process.  The  procedures  in  these 
stages  included: 


1.  Researchers  at  ARI  reviewed  the  questionnaire  contents 
used  in  the  1984  and  1985  New  Recruit  Surveys  for  clarity, 
necessity  of  content  to  support  ongoing  in-house  research 
efforts,  and  the  accuracy  of  content  in  response  to  policy 
changes  in  order  to  construct  the  1986  item  bank.  Those  items 
that  were  dated  due  to  changes  in  policy  and/or  programs  were 
dropped  from  the  item  bank.  Previously  used  questionnaire  items 
were  rewritten  as  necessary  to  improve  their  readability, 
response  availability,  and  accuracy.  New  items  were  included  in 
the  1986  and  1987  item  banks  to  meet  the  needs  of  existing  and 
anticipated  research  programs. 

2.  The  1986  item  bank,  developed  by  ARI,  was  then  forwarded 
to  the  NRS  sponsor  (the  U.S.  Army  Recruiting  Command)  and  to 
users  of  NRS  survey  data  (Office  of  the  Deputy  Chief  of  Staff  for 
Personnel;  Reserve  Officer  Training  Corps;  National  Guard  Bureau; 
and  Office  of  the  Chief  of  Army  Reserve)  for  their  review. 
Reviewers  were  asked  to  prioritize  the  entire  survey  item  bank  by 
determining  for  each  item  whether  it  (a)  must  be  included,  (b) 
must  be  included  if  the  1986  survey  were  to  be  reused  in  1987, 

(c)  should  be  included  but  could  be  omitted  for  both  1986  and 
1987,  and  (d)  need  not  be  asked  but  could  be  used  for  segmenting 
responses  of  more  high-priority  items. 

3.  Based  on  the  priorities  assigned,  a  final  item  bank  was 
assembled  and  provided  to  USAREC  for  review  and  approval.  This 
item  bank  was  then  transmitted  to  Data  Recognition  Corporation 
for  their  use  in  developing  the  survey  questionnaire  booklets. 


Survey  Planning  and  Administration  Difficulties 

In  the  1984  and  1985  recommendations  for  future 
administrations  of  the  New  Recruit  Surveys  (Celeste,  Wilson, 
Ramsey,  Elig,  &  Pliske,  1986)  it  was  suggested  that  an  item 
analysis  be  conducted  prior  to  developing  any  future  NRS  surveys. 
They  also  recommended  that  the  validity  of  survey  items  be 
determined  prior  to  their  use  in  the  NRS  surveys,  and  they  raised 
questions  about  the  consistency  and  missing  values  contained  in 
the  surveys.  Not  the  least  of  the  contractor's  recommendations 
was  that  the  final  survey  content  be  complete  at  least  one-month 
before  the  first  administration  date. 

None  of  the  contractor's  recommendations  were  implemented. 
The  recommendations  were  published  in  the  same  month  that  the 
1986  surveys  were  being  printed.  The  1986  NRS  instrument  was 
still  undergoing  content  additions  and  changes  as  late  as  five 
days  before  the  first  administration  date.  The  development  of 
the  1987  USAREC  NRS  was  also  slowed  by  procurement  delays  and  as 
a  result  the  same  instrument,  with  minor  corrections  to  original 
proofs,  was  used.  Further,  no  item  analyses  were  conducted  on 
any  of  the  NRS  survey  series. 


Across  the  years,  the  New  Recruit  surveys  have  increased  in 
terms  of:  (1)  number  of  questions  asked,  (2)  number  of  responses 
possible  per  question,  (3)  number  of  variables  coded  onto  the 
database,  (4)  reading  difficulty  level,  and  (5)  survey  form 
complexity. 

The  1982  Original  Form  of  the  DA  Survey  of  Personnel 
Entering  the  Army  consisted  of  80  questions  with  211  variables 
entered  onto  the  database.  The  Revised  Form  of  the  1982  Survey 
consisted  of  239  variables.  The  overall  reading  difficulty  level 
of  the  1982  series  of  surveys  averaged  at  the  six  grade  level. 

The  1983  Summer  Survey  contained  from  158  to  160  questions  that 
yielded  651  variables  with  a  reading  level  equal  to  the  1982 
series.  In  the  Summer  of  1984,  recruits  answered  either  150  or 
160  questions  with  a  resulting  variable  count  of  556  and  an 
increase  in  reading  level  to  the  8th  grade.  In  the  Summer  of 
1985  the  respondents  were  asked  from  165  to  191  questions 
yielding  771  variables  and  a  reading  level  of  some  college. 
Finally,  the  1986  and  1987  recruit  respondents  were  asked  230 
questions  and  the  recorded  variables  reached  863  and  862 
respectively.  While  a  number  of  the  variables  were  created 
during  database  processing  and  creation,  the  majority  were  the 
result  of  increased  numbers  of  response  categories  and  the  almost 
tripling  of  the  number  of  questions  since  1982. 

Despite  the  continual  increase  in  the  number  of  questions 
and  response  alternatives,  the  time  frame  for  completing  the  NRS 
series  of  surveys  has  remained  relatively  constant.  Recruits 
have  been  given  approximately  one  (1)  hour  to  complete  the 
surveys.  In  1982,  with  a  maximum  of  80  questions  and  211 
variables,  this  allowed  for  approximately  45  seconds  to  read, 
evaluate,  and  respond  to  each  question.  In  1987,  with  230 
questions,  this  reduced  the  time  allowed  to  less  than  15  seconds 
per  question.  Even  with  innovative  skip  patterns  built  into  the 
instrument,  the  quality  of  a  response  generated  in  such  a  brief 
time  frame  is  questionable. 

Implications  for  Survey  Data  Users 

These  dramatic  increases  in  the  number  of  questions  recruits 
are  asked  to  answer  suggest  that  the  New  Recruit  Surveys  may  no 
longer  be  maximizing  their  function  as  program  evaluation  and 
evaluation  instruments.  Instead,  the  ever  increasing  length  and 
complexity  may  have  changed  the  instrument  into  a  form  of  what 
psychometricians  call  a  power  test.  In  the  classic  power  test, 
respondents  must  complete  all  sections  of  a  test  instrument  in 
order  to  achieve  the  maximum  possible  score.  Such  instruments 
are  deliberately  designed  to  favor  the  speediest  respondent  with 
the  highest  ability. 


The  New  Recruit  Survey  represents  a  type  of  instrument  whose 
sole  function  is  to  collect  information  that  will  be  analyzed  as 
group  data.  There  is  no  intent  to  select  out  individuals  on  the 
basis  of  their  response  acceptability  or  their  success  in 
completing  the  questionnaire.  Because  of  the  length  of  the 
questionnaire,  it  is  possible  that  responses  for  certain 
categories  of  respondents  are  under-represented  in  later 
sections  of  the  NRS.  Such  respondents  could  include  recruits  who 
differ  along  a  range  of  personal  and  demographic  variables. 


PROCEDURE  AND  RESULTS 


Readability  Analysis 

A  readability  analysis  (Flesch,  1974;  1949)  was  conducted  on 
the  four  separate  sections  of  the  1987  New  Recruit  Survey. 

Samples  of  text  were  randomly  selected  from  each  questionnaire 
section  and  the  number  of  sentences,  sentence  length,  and  number 
of  syllables  were  evaluated.  The  readability  results  indicate 
that  those  items  encountered  by  the  recruits  in  the  first  50 
questions  of  the  first  section  (Your  Enlistment)  should  have  been 
easily  readable  by  respondents  with  reading  skills  at  the  6th- 
grade  level.  Further  into  the  first  section,  the  reading 
difficulty  increases  above  the  8th-grade  level,  at  the  same  point 
as  more  complex  skip  patterns  are  encountered  (approximately 
Question  90) .  Survey  items  in  the  second  section  (Leisure  Time 
Activities) ,  which  are  approximately  150  questions  into  the 
survey,  require  a  reading  level  of  at  least  high  school  or  some 
college. 

Survey  Length  And  Selected  Demographic  Analyses 

Analyses  of  the  last  survey  question  completed  were 
conducted  on  data  from  the  Summer  1987  USAREC  New  Recruit  Survey 
by  Test  Category  (as  computed  from  ASVAB  raw  sub-test  scores) , 
ethnic  group  membership,  rural/urban  nature  of  residence  before 
entering  service,  gender,  education  level,  and  term  of 
enlistment.  If  no  consistent  bias  was  being  introduced  because 
of  survey  length,  then  it  would  be  expected  that  no  difference  in 
the  completion  rates  of  the  survey  should  exist  among 
subcategories  of  each  variable.  An  analysis  of  the  mean  number 
of  questions  answered  for  those  respondents  who  did  not  complete 
the  survey  shows  that  the  mean  survey  item  completed  was  #205. 

Respondent  Gender.  The  completion  rate  of  female  recruits 
(86.6%)  exceeded  that  of  males  (78.5%).  Female  respondents 
completed  significantly  more  survey  items  than  males  (M  =  227.4 
versus  M  =  224.6).  However,  it  is  possible  that  this  difference 
may  be  explained  by  the  qualifications  for  female  recruits  being 
more  rigorous  in  terms  of  AFQT  and  educational  level,  factors 
that  are  discussed  separately. 


Educational  Level.  As  was  expected  from  the  reading  difficulty 
level  analysis,  more  recruits  with  some  post-secondary  attendance 
(88.2%)  completed  the  survey  than  high  school  diploma  graduates 
(79.3%).  Non-high  school  diploma  graduates  completed  the  entire 
survey  least  often  (75.8%).  Significant  differences  exist 
between  these  groups  in  the  mean  number  of  questions  answered  as 
indicated  by  analysis  of  variance.  F(2,  2996)  =  4.96,  p  <  .007. 
Evaluation  of  the  separate  means  by  the  most  conservative 
Scheffe  Test  indicates  that  post-high  school  diploma  graduate 
recruits  were  the  most  successful  at  completing  the  surveys  (M  = 
228.5)  followed  by  high  school  diploma  graduates  (M  =  224.9)  and 
answered  significantly  more  items  than  non-high  school  diploma 
graduates  (M  =  221.8). 

AFOT  Test  Category.  There  are  marked  differences  in  the 
completion  rates  among  Test  Category  I,  II,  IIIA,  and  IIIB/IV 
respondents.  Eighty-eight  percent  (88.1%)  of  Category  I  recruits 
completed  the  entire  questionnaire,  compared  to  Category  II 
(79.2%),  Category  IIIA  (72.1%)  and  Category  IIIB/IV  (58.2%) 
recruits.  Significant  differences  also  exist  between  these 
groups  in  the  mean  number  of  questions  answered  as  indicated  by 
analysis  of  variance.  F(5,  2994)  =  99999.99,  p  <  .001.  Scheffe 
Test  results  indicate  that  Category  I  respondents  were  the  most 
successful  at  completing  the  surveys  (M  =  228.9)  and  answered 
significantly  more  items  than  either  Category  IIIA's  (M  =  225.5), 
IIIB's  (M  =  222.5),  or  IV' s  (M  =  217.5). 

Ethnic  Group  Membership.  A  higher  percentage  of  white  survey 
respondents  completed  the  questionnaire  (82.8%)  than  Hispanics 
(76.2%),  Blacks  (71.8%),  or  Other  (American  Indian,  Alaskan 
Native,  Asian,  and  Pacific  Islander)  (69.6%)  ethnic  groups. 
However,  after  controlling  for  the  effects  of  Test  Category,  no 
significant  differences  exist  among  the  ethnic  groups  average 
number  of  questions  answered  on  the  surveys  (White  (M  =  225.7) ; 
Black  (M  =  224.7);  Hispanic  (M  =  222.8);  and  Other  (M  =  222.4). 

Enlistment  Term.  Type  of  Incentive.  Recruiting  Brigade. 

Rural /Urban  Residence.  No  reportable  differences  in  completion 
rates  were  noted  for;  (1)  2-,  3-,  or  4 -year  enlistments,  (2)  Army 
College  Fund  or  Cash  enlistment  bonus  recipients,  (3)  recruiting 
brigade  location,  or  (4)  residence  in  rural  versus  urban  settings 
prior  to  enlistment. 


Life  Table  Analyses 

Using  a  life  tabling  technique  to  analyze  the  completion 
rates  of  various  groups  for  the  NRS  survey  yielded  two  sets  of 
results.  The  first  set  support  the  ANOVA  findings  for  the 
effects  of  AFQT  Test  Category.  A  differential  survey  completion 
life  exists  among  the  individuals  of  different  Test  Category. 

That  is,  respondents  who  are  Test  Category  I  complete  more  survey 


items  than  Category  II  respondents,  who  complete  more  items  than 
Category  IIIA  respondents,  etc.  The  opposite  of  this  is  failure 
rate,  where  Category  IV  respondent  complete  fewer  survey  items 
than  Category  IIIB  respondents.  In  other  words,  if  completion  of 
the  entire  questionnaire  is  the  goal,  anything  short  of  this  goal 
is  a  failure.  That  point  in  the  survey  where  the  respondent 
stops  answering  survey  items  is  the  failure  event. 

Table  1  contains  the  percentage  of  the  last  question 
answered  by  respondents  on  the  1987  USAREC  New  Recruit  Survey. 
Based  on  the  percentage  of  failures  in  a  standard  life  table 
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Table  1 

Percentage  of  Respondents  Bv  Category  Failing  to  Complete  All  NRS 
Survey  Items 

Survey  Items  Completed 
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analysis,  each  cell  contains  the  percent  failures  within  each 
Test  Category/Educational  Level  subgroup  during  different 
sections  of  the  survey.  Through  examination,  it  is  clear  that 
many  of  those  respondents  who  did  not  complete  the  entire  survey 
were  working  on  questions  150  to  200  when  either  administration 
time  was  up  or  they  stopped  for  reasons  of  their  own. 


Overall,  only  17  respondents  stopped  answering  before 
Question  #150.  173  respondents  stopped  before  Question  #200,  and 


080 


the  remaining  425  respondents  stopped  before  Question  230.  This 
means  that  approximately  20  percent  of  the  recruits  administered 
the  1987  New  Recruit  Survey  never  reached  the  final  item. 


Test  Length  And  Cumulative  Completion  Rates 

The  graphed  cumulative  percentage  of  the  last  question 
completed  by  Test  Category  is  presented  in  Figure  1.  An 
examination  of  the  graphed  percentages  shows  that  test  length 
begins  to  effect  completion  rates  for  all  respondents,  but 
particularly  Category  IIIB/IV  respondents  after  approximately  150 
questions.  By  1986  and  1987  survey  booklet  standards,  recruits 
reaching  this  point  in  the  survey  would  have  answered  those 
questions  related  to  their  enlistment  activities.  That  means 
that  many  recruit  respondents  would  not  get  to  sections 
containing  media,  background,  and  personal  demographic  related 
questions . 

For  program  evaluation  and  analysis  purposes,  continued  use 
of  a  USAREC  New  Recruit  Survey  containing  more  than  150  questions 
of  the  same  type  and  reading  difficulty  as  exists  in  the  1987 
questionnaire  raises  a  question  of  bias  and  representativeness. 

If  only  responses  from  Test  Category  I  to  IIIA  respondents  are 
desired,  then  a  survey  containing  up  to  200  of  the  present  items 
may  be  ideal.  However,  if  comparisons  are  planned  to  other 
groups  (such  as  Test  Category  IIIB's  and  IV' s)  there  is  a  strong 
possibility  that  those  questionnaire  items  that  are  placed  toward 
the  end  of  the  survey  will  be  biased.  Many  Category  IIIB  and  IV 
respondents  will  not  get  that  far  into  the  survey.  As  an 
effective  data  collection  instrument  for  all  types  of 
respondents,  the  present  survey's  length  and  reading  difficulty 
levels  are  too  great. 


RECOMMENDATIONS  FOR  NRS-88  AND  BEYOND 

Based  on  the  analyses  conducted  on  the  1987  USAREC  Survey  of 
Active  Army  Recruits,  the  following  recommendations  should  be 
incorporated  into  future  New  Recruit  Survey  development  and 
administrations  as  well  as  serving  as  a  reminder  to  those  who  may 
be  called  upon  to  implement  any  such  surveys. 

1.  Decrease  Survey  Content.  The  New  Recruit  Survey  series 
was  designed  to  provide  research  program  analysis  and  evaluation 
data  for  use  by  the  U.S.  Army.  It  was  not  intended  to  serve  as  a 
screening  test  for  AFQT  or  educational  level .  The  instrument 
should  be  shortened  to  a  maximum  length  of  150  items  and  as  a 
second  measure,  a  page  total  cap  of  12  single  sheets  should  be 
followed.  Even  with  such  shortening  of  length,  the 
recommendations  that  follow  should  be  implemented. 


2.  Pretest  the  Survey  Instrument.  The  New  Recruit  Surveys 
should  be  pretested  on  a  representative  of  group  of  respondents 
(approximately  100)  to  ensure  the  questionnaire  can  be  completed 
by  most  recruit  respondents.  The  author  has  followed  the  rule 
that  95%  or  more  of  the  respondents  should  be  able  to  complete  a 
survey  used  for  program  analysis  and  evaluation  purposes  in  the 
allotted  time  frame.  The  pretest  should  consider  both  survey 
length  and  item  understanding.  We  have  seen  that  length 
considerations  occur  for  different  groups,  but  a  survey  must  also 
be  written  at  a  reading  and  understanding  level  commensurate  with 
the  respondents'  abilities.  To  insure  that  reading  level 
difficulties  are  minimized,  the  questionnaire  reading  difficulty 
level  should  not  exceed  the  6th-grade  level.  This  can  be 
accomplished,  in  part,  by  using  shortened  question  stems  and 
response  options,  as  well  as  by  reducing  the  use  of  multi¬ 
syllabic  words. 

3.  Explore  Alternatives  to  Skip  Patterns.  Imbedded  skip 
patterns  serve  the  purpose  of  reducing  respondent  burden  for 
marking  inapplicable  responses  as  well  as  increasing  the 
effective  size  of  the  instrument.  However,  imbedded  skip 
patterns  introduced  into  an  already  onerous  questionnaire  make 
reading  and  following  the  flow  of  items  difficult.  Care  should 
be  taken  to  reduce  the  use  of  imbedded  skip  patterns  to  the 
minimum  necessary  to  collect  the  needed  information.  Larger 
highlighting  of  skip  patterns  areas,  the  use  of  larger  and  more 
colorful  key  item  screening,  and  the  design  of  survey  booklets  to 
insure  an  imbedded  skip  pattern  is  contained  on  a  single  page  may 
reduce  errors  associated  with  incorrectly  entering  the  patterns. 

4.  Increase  Administration  Time  Requests.  One  factor  that 
can  be  controlled  at  the  administration  level  is  time.  When 
requesting  troop  support  for  future  NRS  administrations,  insure 
that  the  request  includes  sufficient  time  for  troop  movements  and 
both  administration  instructions  and  a  full  time  allotment  in 
which  to  complete  the  survey. 

5.  Increase  Survey  Preparation  Time.  One  of  the  most 
common  difficulties  associated  with  the  NRS  effort  has  been 
delays  caused  by  late  contracting,  poor  response  times  in 
evaluating  item  banks,  and  schedule  disruptions  caused  by  last- 
minute  questionnaire  changes  or  additions.  The  final  item  pool 
for  a  main  survey  instrument  should  be  approved  at  least  six  (6) 
months  prior  to  administration.  Pretests  should  occur  at  least 
3  months  before  administration.  Final  survey  printing  should 
occur  at  least  one  (1)  month  before  field  administration. 

Without  such  prior  planning,  short-cuts  will  and  do  occur. 
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